Tool bit holders are known that typically consist of a member that has one end that is retained in the chuck of a hand or powered rotary tool such as a drill, power driver, screw driver or other rotary tool. The other end of the member supports a bit holder that releasably holds a tool bit such as a screw driver bit, drill bit or the like. The tool bit holder is retained in the chuck of the rotary tool such that the bits can be quickly inserted into the holder without having to manipulate the chuck. Tool bit holders are known to use a ball to retain a quick change style bit in the holder. These tool bit holders require two-hand removal of the bit and only work with quick change style bits.